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US  ARMY  PROCUREMENT  RESEARCH  OFFICE 


EXECUTIVE  SUMMARY 


BACKGROUND 

This  report  describes  the  latest  in  a series  of  tests  the  Army  Procurement 
Research  Office  (APRO)  is  doing  to  make  the  source  selection  technique.  Computer 
Assisted  Total  Value  Assessment  (CATVA),  operationally  suitable  for  use  in  DARCOM. 
This  test  for  the  first  time  applies  CATVA  to  an  on-going  source  selection. 


OBJECTIVE 

TRe  specific  objectives  of  this  effort  are  to  see  if  the  CATVA  model, 
heretofore  merely  feasible,  can  be  of  significant  use  in  aiding  an  actual  source 
selection  decision  and  at  the  same  time  to  seek  more  improvements  to  the  model. 


METHODOLOGY 

The  test  was  done  by  instructing  source  selection  personnel  on  the  use  of 
CATVA,  letting  them  use  CATVA  in  the  source  selection  and  then  evaluating  its  use.? 


FINDINGS 

'•  CATVA  was  useful  as  a backup  tool  in  the  source  selection  decision. 


2.  It  was  particularly  useful  in  planning  for  the  decision  and  in  making 
sensitivity  analyses  on  the  impact  of  varying  assumptions  and  systems  configur- 
ations on  proposal  rankings. 


3.  CATVA  did  discriminate  well  in  early  runs,  but  lost  considerable  discrimin- 
ating power  in  later  ones,  probably  due  to  technical  leveling. 


4.  Source  selection  management  can  avoid  or  minimize  evaluator  resistance  to 
the  rigorous  CATVA  analysis  by  requiring  them  to  give  the  evaluated  data  in  a 
proper  format,  without  involving  them  in  CATVA  operation. 


5.  CATVA  can  handle  non-linear  utility  (without  reprogramming)  by  manual  con- 
version before  entering  data  into  the  computer  (see  Page  4). 


RECOWENOATION 

Distribute  this  report  by  letter  from  Procurement  and  Production  Directorate 
DARCOM,  making  complete  references  to  all  previous  reports  and  suggesting 
each  PM  and  major  Command  consider  the  technique  for  application.  Indicate 
that  APRO  will  assist  in  implementation. 
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CHAPTER  I 


INTRODUCTION 

BACKGROUND 

This  report  describes  the  latest  in  a series  of  tests  of  the  Computer 
Assisted  Total  Value  Assessment  (CATVA)  technique  developed  by  the  Army 
Procurement  Research  Office  (APRO)  for  use  in  US  Army  Materiel  Development  and 
Readiness  Command  (DARCOM)  source  selections.  The  first  effort  was  on  a simu- 
lated source  selection  to  assess  the  theoretical  feasibility  of  the  technique. 
The  second  test  was  a completed  major  source  selection;  that  is,  the  research 
team  used  after-the-fact  source  selection  data  (augmented  by  interviews)  to 
see  how  the  technique  would  work  on  a real  procurement. 

This  test  involves  the  actual  use  of  CATVA  on  an  on-going  selection,  the 
XM1  engineering  development  procurement.  It  is  the  latest  step  in  APRO's  plan 
to  help  DARCOM  make  CATVA  operationally  suitable. 

REVIEW  OF  CATVA  OPERATION 

For  readers  not  having  seen  APRO  Report  501-1  and  APRO  Report  501-2, 
a short  review  of  CATVA ‘s  operation  is  in  order. 

CATVA  is  based  on  the  concept  of  value;  that  is,  the  best  proposal 
will  be  that  one  giving  the  "best"  combination  of  performance,  management, 
and  cost  features.  CATVA  assesses  the  total  value  of  proposal  through  a 
rigorous  analysis  of  only  the  measurable  (or  scalable  by  adjectival  gradations) 
performance,  management,  and  cost  parameters  of  the  proposed  system.  Each 
performance,  management  and  cost  parameter  is  an  evaluation  factor  in  the 


proposal  evaluation;  the  value  of  each  factor  is  based  on  a comparison  of 
the  contractor's  response  to  what  the  Government  requires. 

The  CATVA  technique  features  computerized  processing  of  the  value 
assessments  and  a computer  graphics  display  of  these  values.  Included  in 
CATVA's  computerized  processing  is  a statistical  discrimination  routine  which 
aids  the  evaluators  in  discriminating  among  the  proposals.  This  routine 
identifies  the  best  proposal (s),  based  on  the  total  value  expression,  by 
calculating  a competitive  range  for  various  levels  of  confidence.  The 
graphical  display  is  unique  to  the  CATVA  concept.  It  is  a dynamic  represen- 
tation of  the  individual  value  scores  for  the  decision  maker  who  with  it  can 
discern  the  relative  strengths  and  weaknesses  of  proposals  in  a rapid  and 
efficient  manner.  The  CATVA  model  is  based  on  a spatial  analogy  between  a 
contractor's  proposal  and  the  total  value  of  his  proposal  such  that  the  larger 
the  area  representing  a contractor's  proposal  the  larger  his  proposal  valued 
RESULTS  OF  TESTS 

The  test  on  the  simulated  source  selection  showed  that  the  CATVA  concept 
was  feasible.  A few  minor  programing  adjustments  were  found  to  be  necessary 
however. 
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The  after-the-fact  test  was  also  reasonably  successful.  CATVA  was 
useable  and  did  discriminate.  There  were,  of  course,  inherent  problems  with 
using  a source  selection  not  designed  for  CATVA,  but  they  were  felt  to  be 
truly  peculiar  to  the  test.  Longer  term  difficulties  identified  include: 
some  original  CATVA  design  specs  limited  scope  of  technique  (i.e.,  independent 
factors,  linear  utility,  deterministic  in  nature);  some  data  such  as  RAM-D, 
is  highly  unreliable  for  estimation;  evaluators  do  not  have  a high  level  of 
confidence  in  their  estimates;  and  source  selection  personnel  perceive  that 
an  automated  system  limits  or  inhibits  their  individual  judgement  and  discretion. 

The  text  that  follows  describes  the  results  of  the  XM1  source  selection 
test. 


CHAPTER  II 


CHARACTERISTICS  OF  THE  XM1  CATVA  TEST 


The  XM1  CATVA  test*  differed  from  previous  tests  in  that: 

1.  Extensive  sensitivity  analysis  was  done  to  see  effects  of  different 
score  and  weight  inputs. 

2.  Scores  were  manually  bounded  high  and  low;  that  is,  after  a certain 
point,  an  increase  in  a factor  value  received  no  more  score,  or  a decrease 
received  no  less  score,  as  shown  in  Figure  1.  This  is,  in  effect,  a manual 
solution  to  the  measurement  of  non-linear  utility. 


UNITS  OF  PERFORMANCE  ($,  MPH,  ETC) 


EXAMPLE  OF  BOUNDED  SCORE 
FIGURE  1 


*A  detailed  description  of  CATVA's  use  during  the  XM1  source  selection  is 
provided  in  references  3 and  4.  These  documents  are  classified  and  sensitive 
and  may  be  available  to  authorized  personnel  through  the  XM1  Project  Office, 
Universal  City  Professional  Building,  28150  Dequindre,  Warren,  Michigan  48092 
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3.  An  abstract  "benchmark"  was  put  into  the  computer  to  indicate 
desired  level  of  performance.  In  Figure  1 for  example,  if  .5  were  put  into 
the  computer,  this  would  indicate  400  units  of  performance  were  desired. 

These  benchmarks  were  established  on  a common  scale  for  easy  factor  compari- 

• sons  and1 were  not  required  for  CATVA. 

4.  The  translation  from  actual  performance  to  score  (e.g.,  from  400 
units  to  .5)  was  done  by  hand  and  the  score  then  put  into  the  computer.  This 
was,  of  course,  in  lieu  of  reprogramming  the  basic  CATVA  to  have  the  computer 
make  this  translation. 

5.  The  XM1  resolicitation  brought  on  multiple  proposals  from  the  two 
offerors  and  thereby  allowed  for  the  first  CATVA  multiple  evaluation. 

6.  The  evaluators  themselves  did  not  need  to  be  involved  in  direct  usage 
of  CATVA.  The  SSEB  management  staff  simply  needed  input  in  the  correct 
format  from  individual  evaluators  or  teams. 
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CHAPTER  III 


TEST  RESULTS 


FINDINGS 

1.  One  of  the  main  uses  of  CATVA  was  the  sensitivity  analysis  performed 
on  different  types  of  inputs.  The  iterative  runs  let  decision  makers 

know  quickly  the  impact  of  varying  assumptions  and  system  configurations 
on  the  proposals  rankings.  In  this  way,  CATVA  gave  decision-makers  a lot  of 
insight  into  the  relative  merits  of  the  proposals.  For  this  reason  CATVA  can 
be  just  as  useful  (if  not  more  useful)  in  planning  for  the  source  selection 
decision  as  in  helping  to  actually  make  it. 

2.  Rather  than  reprogramming  CATVA,  evaluators  made  high  and  low 
"cutoff"  scores  (see  Fig  1)  by  manually  converting  evaluated  performance 
figures  into  scores.  This  avoided  the  problem  of  unrealistically  high  and 
low  scores. 

3.  The  resistance  of  evaluators  to  the  CATVA  concept  can  be  minimized  by 
simply  not  involving  them  in  the  technique.  By  asking  for  evaluated  input 

in  the  proper  format,  SSEB  managers  can  use  CATVA  as  a strict  management  tool. 

4.  CATVA  did  discriminate  well  in  early  proposal  evaluation  runs,  but 
lost  considerable  discriminating  power  in  the  later  ones  with  updated  pro- 
posals. It  is  probable  that  considerable  technical  leveling  took  place  as 
time  passed. 

5.  The  concept  is  again  held  to  be  feasible.  Users  said  that  CATVA  was 
useful  particularly  in  the  early  deliberations  in  (i)  organizing  the  source 
selection  data;  (ii)  expressing  each  contractor's  proposal  quantitatively 


with  a total  value  expression;  (iii)  discriminating  among  the  proposals  where 
possible;  (iv)  analyzing  the  sensitivity  of  the  total  value  to  various  factors, 
and  (v)  summarizing  the  results  in  convenient  replicable  displays. 

RECOMMENDATION 

Distribute  this  report  by  letter  from  Procurement  and  Production  Directorate 
DARCOM,  making  complete  references  to  all  previous  reports  and  suggesting 
each  PM  and  major  Command  consider  the  technique  for  application.  Indicate 
that  APRO  will  assist  in  implementation. 
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